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Abstract 


This study compared the summer learning rates of stu- 
dents who attended Houston Independent School District 
(HISD) 2016 summer school to similar students who did 
not attend, for both promoted and retained students. For 
students between 1* and 2™ grade and those between 2"™4 
and 3" grade, summer learning rates were faster for both 
promoted and retained students who attended HISD 
summer school than comparable students who did not 
attend, but these differences were not statistically signif- 
icant, likely due to the small number of students in each 
group. Similarly, promoted students between 4" and 5" 
grade and retained students between 6" and 7" grade who 
attended summer school had faster learning rates than 
comparable students who did not attend summer school, 
but they were not statistically significant. For these stu- 
dents, it was unclear whether HISD summer school was 
beneficial. However, for students between 5" and 6" grade 
and those between 7" and 8" grade, there is evidence that 
HISD summer school was not beneficial for either pro- 
moted or retained students. This is also true for retained 
students between 4" and 5“ grade and promoted students 
between 6" and 7 grade. 


Introduction 


Relative to the academic year, learning rates typically 
slow down in the summer, especially for economically 
disadvantaged students (Entwisle and Alexander 1992; 
Burkam et al. 2004; Downey and Condron 2016). As such, 
various types of summer interventions have been imple- 
mented throughout the country, especially those geared 
toward students from disadvantaged backgrounds who 
experience the most summer learning loss (Cooper et al. 
2000; McEachin et al. 2016). Despite the wide array of 
summer interventions, relatively little is known about 
whether these programs are effective. This study provides 
an assessment of the effectiveness of HISD 2016 summer 
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school, which the district offered both to retained stu- 
dents (which was required and needed for promotion) as 
well as promoted students (which was not required but 
available for summer enrichment). It should be noted that 
promoted students who marginally fail are more likely 
to attend summer school than other promoted students, 
because they tend to be highly recommended by their 
schools/teachers to do so. Since promoted and retained 
students attend HISD summer school for different pur- 
poses, the effectiveness of summer school was examined 
separately for the two groups. 


Research questions 


= Among promoted students, what is the effect of 
attending HISD summer school relative to those who 
did not attend? Do these effects vary by grade level? 


= Among retained students, what is the effect of 
attending HISD summer school relative to those who 
did not attend? Do these effects vary by grade level? 


= Among students who attended HISD summer school, 
how do the summer learning rates of retained students 
compare with promoted students? 


Data 


This study used the 2015-16 and 2016-17 Istation’s 
Indicators of Progress (ISIP), as well as 2015-16 
Promotion Standards Data. The sample was restricted 
to students in grades 1, 2, 4,5, 6, and 7 during the 2015-16 
academic year and followed them through their subse- 
quent grades. The ISIP tests used different grading scales 
for grades 1-3 (usually under 300 points) and grades 4-8 
(usually above 1500 points). As such, students in grade 3 
during the 2015-16 academic year were eliminated due to 
the difficulty in assessing their progress toward grade 4 
because of inconsistent grading scales. 
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The study focused on students’ ISIP reading scores at 
four time points: the beginning of the 2015-16 fall semes- 
ter, the end of the 2015-16 spring semester, the beginning 
of the 2016-17 fall semester, and the end of the 2016-17 
spring semester, so as to capture both academic year and 
summer learning rates. Students were not tested exactly 
on the first day of school nor on the last day of school, 

and test dates varied by school. Students were tested 
several weeks after the start of the school year and several 
weeks before the end of the school year, and the testing 
window was about two weeks long, both in the fall and in 
the spring. This means that the amount of time between 
the spring test date and the fall test date of the following 
academic year was about five months, significantly longer 
than the actual summer break which was only three 
months long. To get a more accurate estimate of summer 
learning, the reading scores were extrapolated to what 
students would have obtained at the beginning and end of 
summer. The dependent variables were students’ extrap- 
olated ISIP overall reading scores at the four time points 
listed above. 


The key independent variable was HISD summer school 
attendance type’ in the summer of 2016, a four-category 
variable (1=promoted & not attend summer school, 2=pro- 
moted & attend summer school, 3=retained & not attend 
summer school, 4=retained & attend summer school). The 
other independent variables included race/ethnicity, gen- 
der, age, economic disadvantage status, limited English 
proficiency, gifted/talented program, magnet program 
status, and special education status.” 


Methods 


A Piecewise Linear Model (PLM) was used to fit an 
individual learning trajectory for each student as he/she 
progressed through two sequential school years and the 
summer in-between. PLM allows the trajectories to differ 
in each phase in terms of their learning rates, while taking 
account of the independent variables listed in the data 
section. The ISIP reading scores were modeled as repeat- 
ed observations (level one) nested within students (level 
two). To take the clustering effects at the school level into 
account, vce(cluster school ID) options in Stata were used. 


1 Whether a student has attended summer school was identified by 
using variables from the Promotion Standards data. Students with 
non-missing information on either summer campus ID, present 
days, or absent days were categorized as a program attendee; 
otherwise, they were categorized as one who did not attend. 

2 These variables were obtained from the Istation’s Indicators of 
Progress (ISIP) data. 


Average summer learning rates were estimated using 
the difference between the test score at the beginning of 
the 2016-17 fall semester and the score at the end of the 
previous 2015-16 spring semester, divided by the number 
of summer months. Controlling for a variety of indepen- 
dent variables, Figures 1-6 illustrate the learning rates 

of students in grades 1, 2, 4, 5, 6, and 7. Models were run 
separately for different grade levels. 


To explore the robustness of the results further, the 

effect of HISD summer school was estimated using a 
Fuzzy Regression Discontinuity design. The treatment 

of attending HISD summer school can be considered 

as a discontinuous function of students’ overall yearly 
average scores. Students who were just below the criteria 
were mandated to attend the summer program, while 
those who were just above the criteria were not required 
to. The assignment of attending HISD summer school 
was obeyed quite well but not without exceptions, since 
there were a few students who were mandated to attend 
summer school but did not actually attend, and vice 
versa. To deal with the non-compliance issue, a fuzzy 
regression discontinuity design, rather than a sharp 
regression discontinuity, was conducted. Although only 
a small proportion of students did not comply with the 
assignment they received, there might be unobserved 
student attributes that not only caused the noncompliance 
but also affected the outcome to be measured. To solve the 
potential endogeneity (selection into treatment depends 
partially upon starting level of outcome variable) associ- 
ated with participation in summer school, instrumental 
variables—the assignment of attending summer school, 
that is, promotion status at the end of the 2015-16 academ- 
ic year—was utilized. To construct a comparable sample, 
the first promotion criterion—overall yearly average score 
of 70—was used as the cut-off point, and the analytic sam- 
ples were restricted to three bandwidths—students who 
fell within 0.2, 0.3, and 0.5 standard deviations of the cut- 
points of the corresponding grade level. The estimation 
of the treatment effect in a fuzzy regression discontinuity 
was implemented by a 2-stage least squares analysis. 


Findings of Piecewise Linear Model 
(see Figures 1-6) 


Figures 1-6 are based on post-estimation from the 
Piecewise Linear Models, which means that they take 
account of the student factors listed in the data section. 
For students between 1* and 2" grade and those between 
24 and 34 grade, summer learning rates were faster for 
both promoted and retained students who attended HISD 
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summer school than comparable students who did not 
attend, but these differences were not statistically signif- 
icant, likely due to the small number of students in each 
group. Similarly, promoted students between 4" and 5 
grade and retained students between 6" and 7" grade who 
attended summer school had faster learning rates than 
comparable students who did not attend summer school, 
but they were not statistically significant. For these 
students, it is unclear whether HISD summer school was 
beneficial. However, for students between 5" and 6 grade 
and those between 7" and 8" grade, there is evidence that 
HISD summer school was not beneficial for either pro- 
moted or retained students. This is also true for retained 
students between 4" and 5‘ grade and promoted students 
between 6" and 7‘ grade. 


Findings of Fuzzy Regression Discontinuity 
(see Table 1) 


Table 1 presents the results from a two-stage least squared 
(2SLS) instrumental variable (IV) approach. The co- 
efficient for summer school attendance is an unbiased 
estimate of the local average treatment effect (LATE), 
which measures the difference in reading scores between 
students who were mandated to attend summer school 
and actually did so with similarly achieving students who 
were not required to attend summer school and did not 
participate. The LATEs are not statistically significant at 
the .05 level except for students between 4"* and 5" grade, 
suggesting that for most students, HISD summer school 
participation did not increase their reading scores at the 
beginning of the next academic year. As the bandwidths 
around the thresholds become larger, the statistical (non) 
significance of LATE generally holds, with one exception 
(students between 5" and 6" grade). For students between 
4" and 5" grade, the estimates of LATE remain signifi- 
cant, but become smaller as the bandwidth become broad- 
er. Despite the sign on the attend coefficient switches 
direction as the bandwidth changes for students between 
1“ grade and 2 grade, students between 2" and 3" grade, 
students between 6" and 7'* grade, and those between 7 
and 8" grade, the coefficient remains non-significant. This 
means there is little evidence that HISD summer school 
was beneficial for these students. 


Conclusion 


Overall, the benefits of HISD 2016 summer school are un- 
clear for both promoted and retained students. For both 
promoted and retained students between 5" and 6" grade 


The Effects of HISD Summer School 


RESEARCH BRIEF 


and those between 7" and 8" grade, as well as retained 
students between 4" and 5" grade and promoted students 
between 6" and 7" grade, there is no evidence that HISD 
summer school was beneficial. One speculation for these 
findings involves small sample sizes. The analyses were 
conducted separately by grade level, which might include 
small sample sizes for certain groups that are likely to 
affect the statistical power of the results.3 


An alternative explanation is that the HISD 2016 summer 
program consisted of only 22 days of instruction, which 
might not be long enough for the program to take effect. 
Hazelton et al. (1992) suggested that 35 extra days would 
be needed to produce a noticeable change in student 
achievement, and Roderick et al. (2004) found large gains 
in students’ reading performance from a 30 day summer 
program. Moreover, meta analyses provided stronger ev- 
idence that an instructional duration between 60 and 120 
hours is most suitable for a summer program (Cooper et al. 
2000), and reading programs of between 44 and 84 hours 
in length have the largest effects on reading achievement 
(Lauer et al. 2006). However, simply adding a few extra 
days of instruction is not enough to result in impressive 
changes in student achievement (Cooper 2004). For robust 
changes to take place, extra days of instruction must be 
accompanied by appropriate curriculum, qualified teach- 
ers, and student participation (McEachin et al. 2016). 


Limitations 


1. An important caveat should be kept in mind when 
interpreting results from the fuzzy regression 
discontinuity analyses. Since the analytic sample 
only includes compliers who scored closely around 
the cut-points in each grade level, the local average 
treatment effects identified in this study can not speak 
to students who are non-compliers and scored closely 
around the threshold, nor students who did not score 
around the threshold—either high-performing or 
extremely low-performing. 


2. The available data do not include enough information 
on the features of summer school, such as subject of 
instruction (e.g., reading, math, social studies, etc.), 
length (hours) of instruction per day, test scores during 
summer school, class size, teacher qualifications, 
quality of instruction, etc. 


3 Thesample size for the retained & not attend summer school group 
is quite small, which ranges from 11 to 36 students across the six 
grade levels in this study. 
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Improvements to HISD 2018 
Summer School 


The Interventions Office at Houston Independent School 
District has made the following adjustments to the 2018 sum- 
mer program for elementary and middle school students: 


1. Universal screener results for reading and math were 
added to the promotion criteria. 


2. Teachers were to use screener results to identify 
specific summer learning goals for students based on 
their individual needs. 


3. The Interventions department partnered with 
curriculum to redesign summer courses in a way that 
provides teachers with tools for re-teaching the most 
commonly identified misunderstandings as well as 
tools for differentiation. 


4. The Interventions department also partnered with 
Scholastic to implement the use of literacy kits which 
include instruction based on three components: 
Literacy skills, Engagement skills, and Social- 
emotional development. Teachers were trained on the 
use of these kits based on the grade levels that they 
were assigned for summer. 


5. Teachers were provided with checkpoints and data 
trackers so that they could ensure that students were 
mastering concepts daily according to their specific 
learning profile established by the Grade Placement 
Committee. 


6. Students took the reading and or math screener at the 
end of the summer session to determine if there was 
improvement in their scores as compared to the 
End-of-Year screening window. 


7. The end of summer screening results was added 
as acriterion to consider when Grade Placement 
Committees met to determine promotion status 
for students. 
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Table 1. Effects of Attending HISD Summer School on Student Reading Scores* 
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Bandwidth Sample +.2SD +.3SD +.5SD 
Grade 1° 1.159 —1.266 -1.613 
Grade 2 2.564 -1.851 -2.601 
Grade 4 78,359" 58.58" 26.18* 
Grade 5 —26.319 -51.609 —59.455* 
Grade 6 3.095 -10.058 -9.926 
Grade 7 1.956 19.649 mIROZS 


Note: *p<.05 *p<.01 ***p<.001 


4 The model has controlled for a variety of student characteristics: race/ethnicity, gender, economically disadvantage 


status, limited English proficienty, and special education status. 
5 Grade 1, 2, etc. here refers to students attend/not attend summer school after Grade 1, 2, etc. 
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Figure 1. Reading Scores for Promoted and Retained Students 
(Grade 1 in 2015-16) 


240- 1st grade 2015-2016 summer 2016 2nd grade 2016-2017 promoted & not attend 


promoted & attend 


retained & not attend 
220- 


retained & attend 


200- indicates the slope of attendees is statistically 
different from their non-attending counterparts 

B indicates the slope of attendees is not statistically 
different from their non-attending counterparts 

O indicates the slope of retained attendees is 
statistically different from promoted attendees 

& indicates the slope of retained attendees is not 
statistically different from promoted attendees 


Istation overall reading score 


2015-08-24 2016-05-25 2016-08-22 2017-05-25 


Note: The model has controlled for a variety of student characteristics: race/ethnicity, gender, age, 
economic disadvantage status, limited English proficiency, gifted/talented program, magnet program 
status, and special education status. 


Figure 2. Reading Scores for Promoted and Retained Students 
(Grade 2 in 2015-16) 


promoted & not attend 


Jan 2nd grade 2015-2016 summer 2016 3rd grade 2016-2 


promoted & attend 
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retained & attend 


eee & not attend 


4indicates the slope of attendees is statistically 
different from their non-attending counterparts 

B indicates the slope of attendees is not statistically 
different from their non-attending counterparts 

O indicates the slope of retained attendees is 
statistically different from promoted attendees 

indicates the slope of retained attendees is not 
Statistically different from promoted attendees 


230- 


Istation overall reading score 


210- 
2015-08-24 2016-05-25 2016-08-22 2017-05-25 


Note: The model has controlled for a variety of student characteristics: race/ethnicity, gender, age, 
economic disadvantage status, limited English proficiency, gifted/talented program, magnet program 
status, and special education status. 
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Figure 3. Reading Scores for Promoted and Retained Students 
(Grade 4 in 2015-16) 


promoted & not attend 
, 4th grade 2015-2016 summer 2016 5th grade 2016-20 


promoted & attend 


retained & attend 


——— retained & not attend 


Aindicates the slope of attendees is statistically 
different from their non-attending counterparts 

@indicates the slope of attendees is not statistically 
different from their non-attending counterparts 

O indicates the slope of retained attendees is 
statistically different from promoted attendees 

indicates the slope of retained attendees is not 
statistically different from promoted attendees 


2015-08-24 2016-05-25 2016-08-22 2017-05-25 


Note: The model has controlled for a variety of student characteristics: race/ethnicity, gender, age, 
economic disadvantage status, limited English proficiency, gifted/talented program, magnet program 
status, and special education status. 


Figure 4. Reading Scores for Promoted and Retained Students 
(Grade 5 in 2015-16) 


5th grade 2015-2016 summer 2016 6th grade 2016-20 promoted & not attend 


promoted & attend 


retained & attend 


Aindicates the slope of attendees is statistically 
different from their non-attending counterparts 

indicates the slope of attendees is not statistically 
different from their non-attending counterparts 

O indicates the slope of retained attendees is 
Statistically different from promoted attendees 


—— 


es indicates the slope of retained attendees is not 
statistically different from promoted attendees 


2015-08-24 2016-05-25 2016-08-22 2017-05-25 


Note: The model has controlled for a variety of student characteristics: race/ethnicity, gender, age, 
economic disadvantage status, limited English proficiency, gifted/talented program, magnet program 
status, and special education status. 
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Figure 5. Reading Scores for Promoted and Retained Students 
vagy (2Fade 6 in 2015-16) 
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indicates the slope of retained attendees is not 
statistically different from promoted attendees 
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Note: The model has controlled for a variety of student characteristics: race/ethnicity, gender, age, 
economic disadvantage status, limited English proficiency, gifted/talented program, magnet program 
status, and special education status. 


Figure 6. Reading Scores for Promoted and Retained Students 
(Grade 7 in 2015-16) 
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Note: The model has controlled for a variety of student characteristics: race/ethnicity, gender, age, 
economic disadvantage status, limited English proficiency, gifted/talented program, magnet program 
status, and special education status. 
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